% Copyright (c) 2016-2017 Inria.  All rights reserved.
% $COPYRIGHT$
%
% Additional copyrights may follow
%
% $HEADER$

\documentclass[notitlepage]{article}

\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[a4paper]{geometry}
\usepackage{verbatim}
\usepackage{dirtree}

\title{How to use Open~MPI monitoring component}

\author{C. FOYER - INRIA}

\newcommand{\mpit}[1]{\textit{MPI\_Tool#1}}
\newcommand{\ompi}[0]{Open~MPI}
\newcommand{\brkunds}[0]{\allowbreak\_}

\begin{document}

\maketitle

\section{Introduction}

\mpit{} is a concept introduced in the MPI-3 standard. It allows MPI
developers, or third party, to offer a portable interface to different
tools. These tools may be used to monitor application, measure its
performances, or profile it. \mpit{} is an interface that ease the
addition of external functions to a MPI library. It also allows the
user to control and monitor given internal variables of the runtime
system.

The present document is here to introduce the use the \mpit{}
interface from a user point of view, and to facilitate the usage of
the \ompi{} monitoring component. This component allows for
precisely recording the message exchanges between nodes during MPI
applications execution. The number of messages and the amount of data
exchanged are recorded, including or excluding internal communications
(such as those generated by the implementation of the collective
algorithms).

This component offers two types of monitoring, whether the user wants
a fine control over the monitoring, or just an overall view of the
messages. Moreover, the fine control allows the user to access the
results through the application, and let him reset the variables when
needed. The fine control is achieved via the \mpit{} interface, which
needs the code to be adapted by adding a specific initialization
function. However, the basic overall monitoring is achieved without
any modification of the application code.

Whether you are using one version or the other, the monitoring need to
be enabled with parameters added when calling \texttt{mpiexec}, or
globally on your \ompi{} MCA configuration file
(\${HOME}/openmpi/mca-param.conf).  Three new parameters have been
introduced:
\begin{description}
\item [\texttt{-{}-mca pml\brkunds{}monitoring\brkunds{}enable value}]
  This parameter sets the monitoring mode. \texttt{value} may be:
  \begin{description}
  \item [0] monitoring is disabled
  \item [1] monitoring is enabled, with no distinction between user
    issued and library issued messages.
  \item [$\ge$ 2] monitoring enabled, with a distinction between
    messages issued from the library ({\bf internal}) and messages
    issued from the user ({\bf external}).
  \end{description}
\item [\texttt{-{}-mca
    pml\brkunds{}monitoring\brkunds{}enable\brkunds{}output value}]
  This parameter enables the automatic flushing of monitored values
  during the call to \texttt{MPI\brkunds{}Finalize}. {\bf This option
    is to be used only without \mpit{}, or with \texttt{value} =
    0}. \texttt{value} may be:
  \begin{description}
  \item [0] final output flushing is disable
  \item [1] final output flushing is done in the standard output
    stream (\texttt{stdout})
  \item [2] final output flushing is done in the error output stream
    (\texttt{stderr})
  \item [$\ge$ 3] final output flushing is done in the file which name
    is given with the
    \texttt{pml\brkunds{}monitoring\brkunds{}filename} parameter.
  \end{description}
  Each MPI process flushes its recorded data. The pieces of
  information can be aggregated whether with the use of PMPI (see
  Section~\ref{subsec:ldpreload}) or with the distributed script {\it
    test/monitoring/profile2mat.pl}.
\item [\texttt{-{}-mca pml\brkunds{}monitoring\brkunds{}filename
    filename}] Set the file where to flush the resulting output from
  monitoring. The output is a communication matrix of both the number
  of messages and the total size of exchanged data between each couple
  of nodes. This parameter is needed if
  \texttt{pml\brkunds{}monitoring\brkunds{}enable\brkunds{}output}
  $\ge$ 3.
\end{description}


Also, in order to run an application without some monitoring enabled,
you need to add the following parameters at mpiexec time:
\begin{description}
\item [\texttt{-{}-mca pml \^{}monitoring}] This parameter disable the
  monitoring component of the PML framework
\item [\texttt{-{}-mca osc \^{}monitoring}] This parameter disable the
  monitoring component of the OSC framework
\item [\texttt{-{}-mca coll \^{}monitoring}] This parameter disable
  the monitoring component of the COLL framework
\end{description}

\section{Without \mpit{}}

This mode should be used to monitor the whole application from its
start until its end. It is defined such as you can record the amount
of communications without any code modification.

In order to do so, you have to get \ompi{} compiled with monitoring
enabled. When you launch your application, you need to set the
parameter \texttt{pml\brkunds{}monitoring\brkunds{}enable} to a value
$> 0$, and, if
\texttt{pml\brkunds{}monitoring\brkunds{}enable\brkunds{}output} $\ge$
3, to set the \texttt{pml\brkunds{}monitoring\brkunds{}filename}
parameter to a proper filename, which path must exists.

\section{With \mpit{}}

This section explains how to monitor your applications with the use
of \mpit{}.

\subsection{How it works}

\mpit{} is a layer that is added to the standard MPI
implementation. As such, it must be noted first that it may have an
impact to the performances.

As these functionality are orthogonal to the core ones, \mpit{}
initialization and finalization are independent from MPI's one. There
is no restriction regarding the order or the different calls. Also,
the \mpit{} interface initialization function can be called more than
once within the execution, as long as the finalize function is called
as many times.

\mpit{} introduces two types of variables, \textit{control variables}
and \textit{performance variables}. These variables will be referred
to respectively as \textit{cvar} and \textit{pvar}. The variables can
be used to tune dynamically the application to fit best the needs of
the application. They are defined by the library (or by the external
component), and accessed with the given accessors functions, specified
in the standard. The variables are named uniquely through the
application. Every variable, once defined and registered within the
MPI engine, is given an index that will not change during the entire
execution.

Same as for the monitoring without \mpit{}, you need to start your
application with the control variable
\textit{pml\brkunds{}monitoring\brkunds{}enable} properly set. Even
though, it is not required, you can also add for your command line the
desired filename to flush the monitoring output. As long as no
filename is provided, no output can be generated.

\subsection{Initialization}

The initialization is made by a call to
\texttt{MPI\brkunds{}T\brkunds{}init\brkunds{}thread}.  This function
takes two parameters. The first one is the desired level of thread
support, the second one is the provided level of thread support. It
has the same semantic as the
\texttt{MPI\brkunds{}Init\brkunds{}thread} function. Please note that
the first function to be called (between
\texttt{MPI\brkunds{}T\brkunds{}init\brkunds{}thread} and
\texttt{MPI\brkunds{}Init\brkunds{}thread}) may influence the second
one for the provided level of thread support. This function goal is to
initialize control and performance variables.

But, in order to use the performance variables within one context
without influencing the one from an other context, a variable has to
be bound to a session. To create a session, you have to call
\texttt{MPI\brkunds{}T\brkunds{}pvar\brkunds{}session\brkunds{}create}
in order to initialize a session.

In addition to the binding of a session, a performance variable may
also depend on a MPI object. For example, the
\textit{pml\brkunds{}monitoring\brkunds{}flush} variable needs to be
bound to a communicator. In order to do so, you need to use the
\texttt{MPI\brkunds{}T\brkunds{}pvar\brkunds{}handle\brkunds{}alloc}
function, which takes as parameters the used session, the id of the
variable, the MPI object
(i.e. \texttt{MPI\brkunds{}COMM\brkunds{}WORLD} in the case of
\textit{pml\brkunds{}monitoring\brkunds{}flush}), the reference to the
performance variable handle and a reference to an integer value. The
last parameter allow the user to receive some additional information
about the variable, or the MPI object bound. As an example, when
binding to the \textit{pml\brkunds{}monitoring\brkunds{}flush}
performance variable, the last parameter is set to the length of the
current filename used for the flush, if any, and 0 otherwise ; when
binding to the
\textit{pml\brkunds{}monitoring\brkunds{}messages\brkunds{}count}
performance variable, the parameter is set to the size of the size of
bound communicator, as it corresponds to the expected size of the
array (in number of elements) when retrieving the data. This parameter
is used to let the application determines the amount of data to be
returned when reading the performance variables. Please note that the
\textit{handle\brkunds{}alloc} function takes the variable id as
parameter. In order to retrieve this value, you have to call
\texttt{MPI\brkunds{}T\brkunds{}pvar\brkunds{}get\brkunds{}index}
which take as a IN parameter a string that contains the name of the
desired variable.

\subsection{How to use the performance variables}

Some performance variables are defined in the monitoring component:
\begin{description}
\item [\textit{pml\brkunds{}monitoring\brkunds{}flush}] Allow the user
  to define a file where to flush the recorded data.
\item
  [\textit{pml\brkunds{}monitoring\brkunds{}messages\brkunds{}count}]
  Allow the user to access within the application the number of
  messages exchanged through the PML framework with each node from the
  bound communicator (\textit{MPI\brkunds{}Comm}). This variable
  returns an array of number of nodes size typed integers.
\item
  [\textit{pml\brkunds{}monitoring\brkunds{}messages\brkunds{}size}]
  Allow the user to access within the application the amount of data
  exchanged through the PML framework with each node from the bound
  communicator (\textit{MPI\brkunds{}Comm}). This variable returns an
  array of number of nodes size typed integers.
\item
  [\textit{osc\brkunds{}monitoring\brkunds{}messages\brkunds{}sent\brkunds{}count}]
  Allow the user to access within the application the number of
  messages sent through the OSC framework with each node from the
  bound communicator (\textit{MPI\brkunds{}Comm}). This variable
  returns an array of number of nodes size typed integers.
\item
  [\textit{osc\brkunds{}monitoring\brkunds{}messages\brkunds{}sent\brkunds{}size}]
  Allow the user to access within the application the amount of data
  sent through the OSC framework with each node from the bound
  communicator (\textit{MPI\brkunds{}Comm}). This variable returns an
  array of number of nodes size typed integers.
\item
  [\textit{osc\brkunds{}monitoring\brkunds{}messages\brkunds{}recv\brkunds{}count}]
  Allow the user to access within the application the number of
  messages received through the OSC framework with each node from the
  bound communicator (\textit{MPI\brkunds{}Comm}). This variable
  returns an array of number of nodes size typed integers.
\item
  [\textit{osc\brkunds{}monitoring\brkunds{}messages\brkunds{}recv\brkunds{}size}]
  Allow the user to access within the application the amount of data
  received through the OSC framework with each node from the bound
  communicator (\textit{MPI\brkunds{}Comm}). This variable returns an
  array of number of nodes size typed integers.
\item
  [\textit{coll\brkunds{}monitoring\brkunds{}messages\brkunds{}count}]
  Allow the user to access within the application the number of
  messages exchanged through the COLL framework with each node from
  the bound communicator (\textit{MPI\brkunds{}Comm}). This variable
  returns an array of number of nodes size typed integers.
\item
  [\textit{coll\brkunds{}monitoring\brkunds{}messages\brkunds{}size}]
  Allow the user to access within the application the amount of data
  exchanged through the COLL framework with each node from the bound
  communicator (\textit{MPI\brkunds{}Comm}). This variable returns an
  array of number of nodes size typed integers.
\item [\textit{coll\brkunds{}monitoring\brkunds{}o2a\brkunds{}count}]
  Allow the user to access within the application the number of
  one-to-all collective operations across the bound communicator
  (\textit{MPI\brkunds{}Comm}) where the process was defined as
  root. This variable returns a single size typed integer.
\item [\textit{coll\brkunds{}monitoring\brkunds{}o2a\brkunds{}size}]
  Allow the user to access within the application the amount of data
  sent as one-to-all collective operations across the bound
  communicator (\textit{MPI\brkunds{}Comm}). This variable returns a
  single size typed integers. The communications between a process
  and itself are not taken in account
\item [\textit{coll\brkunds{}monitoring\brkunds{}a2o\brkunds{}count}]
  Allow the user to access within the application the number of
  all-to-one collective operations across the bound communicator
  (\textit{MPI\brkunds{}Comm}) where the process was defined as
  root. This variable returns a single size typed integer.
\item [\textit{coll\brkunds{}monitoring\brkunds{}a2o\brkunds{}size}]
  Allow the user to access within the application the amount of data
  received from all-to-one collective operations across the bound
  communicator (\textit{MPI\brkunds{}Comm}). This variable returns a
  single size typed integers. The communications between a process
  and itself are not taken in account
\item [\textit{coll\brkunds{}monitoring\brkunds{}a2a\brkunds{}count}]
  Allow the user to access within the application the number of
  all-to-all collective operations across the bound communicator
  (\textit{MPI\brkunds{}Comm}). This variable returns a single
  size typed integer.
\item [\textit{coll\brkunds{}monitoring\brkunds{}a2a\brkunds{}size}]
  Allow the user to access within the application the amount of data
  sent as all-to-all collective operations across the bound
  communicator (\textit{MPI\brkunds{}Comm}). This variable returns a
  single size typed integers. The communications between a process
  and itself are not taken in account
\end{description}

In case of uncertainty about how a collective in categorized as, please refer to the list given in Table~\ref{tab:coll-cat}.

\begin{table}
  \begin{center}
    \begin{tabular}{|l|l|l|}
      \hline
      One-To-All & All-To-One & All-To-All \\
      \hline
      MPI\_Bcast     & MPI\_Gather   & MPI\_Allgather \\
      MPI\_Ibcast    & MPI\_Gatherv  & MPI\_Allgatherv \\
      MPI\_Iscatter  & MPI\_Igather  & MPI\_Allreduce \\
      MPI\_Iscatterv & MPI\_Igatherv & MPI\_Alltoall \\
      MPI\_Scatter   & MPI\_Ireduce  & MPI\_Alltoallv \\
      MPI\_Scatterv  & MPI\_Reduce   & MPI\_Alltoallw \\
      && MPI\_Barrier \\
      && MPI\_Exscan \\
      && MPI\_Iallgather \\
      && MPI\_Iallgatherv \\
      && MPI\_Iallreduce \\
      && MPI\_Ialltoall \\
      && MPI\_Ialltoallv \\
      && MPI\_Ialltoallw \\
      && MPI\_Ibarrier \\
      && MPI\_Iexscan \\
      && MPI\_Ineighbor\_allgather \\
      && MPI\_Ineighbor\_allgatherv \\
      && MPI\_Ineighbor\_alltoall \\
      && MPI\_Ineighbor\_alltoallv \\
      && MPI\_Ineighbor\_alltoallw \\
      && MPI\_Ireduce\_scatter \\
      && MPI\_Ireduce\_scatter\_block \\
      && MPI\_Iscan \\
      && MPI\_Neighbor\_allgather \\
      && MPI\_Neighbor\_allgatherv \\
      && MPI\_Neighbor\_alltoall \\
      && MPI\_Neighbor\_alltoallv \\
      && MPI\_Neighbor\_alltoallw \\
      && MPI\_Reduce\_scatter \\
      && MPI\_Reduce\_scatter\_block \\
      && MPI\_Scan \\
      \hline
  \end{tabular}
\end{center}
  \caption{Collective Operations Categorization}
  \label{tab:coll-cat}
\end{table}

Once bound to a session and to the proper MPI object, these variables
may be accessed through a set of given functions. It must be noted
here that each of the functions applied to the different variables
need, in fact, to be called with the handle of the variable.

The first variable may be modified by using the
\texttt{MPI\brkunds{}T\brkunds{}pvar\brkunds{}write} function. The
later variables may be read using
\texttt{MPI\brkunds{}T\brkunds{}pvar\brkunds{}read} but cannot be
written. Stopping the \textit{flush} performance variable, with a call
to \texttt{MPI\brkunds{}T\brkunds{}pvar\brkunds{}stop}, force the
counters to be flushed into the given file, reseting to 0 the counters
at the same time. Also, binding a new handle to the \textit{flush}
variable will reset the counters. Finally, please note that the size
and counter performance variables may overflow for multiple large
amounts of communications.

The monitoring will start on the call to the
\texttt{MPI\brkunds{}T\brkunds{}pvar\brkunds{}start} until the moment
you call the \texttt{MPI\brkunds{}T\brkunds{}pvar\brkunds{}stop}
function.

Once you are done with the different monitoring, you can clean
everything by calling the function
\texttt{MPI\brkunds{}T\brkunds{}pvar\brkunds{}handle\brkunds{}free} to
free the allocated handles,
\texttt{MPI\brkunds{}T\brkunds{}pvar\brkunds{}session\brkunds{}free}
to free the session, and \texttt{MPI\brkunds{}T\brkunds{}Finalize} to
state the end of your use of performance and control variables.

\subsection{Overview of the calls}

To summarize the previous informations, here is the list of available
performance variables, and the outline of the different calls to be
used to properly access monitored data through the \mpit{} interface.
\begin{itemize}
\item \textit{pml\brkunds{}monitoring\brkunds{}flush}
\item
  \textit{pml\brkunds{}monitoring\brkunds{}messages\brkunds{}count}
\item \textit{pml\brkunds{}monitoring\brkunds{}messages\brkunds{}size}
\item
  \textit{osc\brkunds{}monitoring\brkunds{}messages\brkunds{}sent\brkunds{}count}
\item
  \textit{osc\brkunds{}monitoring\brkunds{}messages\brkunds{}sent\brkunds{}size}
\item
  \textit{osc\brkunds{}monitoring\brkunds{}messages\brkunds{}recv\brkunds{}count}
\item
  \textit{osc\brkunds{}monitoring\brkunds{}messages\brkunds{}recv\brkunds{}size}
\item
  \textit{coll\brkunds{}monitoring\brkunds{}messages\brkunds{}count}
\item
  \textit{coll\brkunds{}monitoring\brkunds{}messages\brkunds{}size}
\item \textit{coll\brkunds{}monitoring\brkunds{}o2a\brkunds{}count}
\item \textit{coll\brkunds{}monitoring\brkunds{}o2a\brkunds{}size}
\item \textit{coll\brkunds{}monitoring\brkunds{}a2o\brkunds{}count}
\item \textit{coll\brkunds{}monitoring\brkunds{}a2o\brkunds{}size}
\item \textit{coll\brkunds{}monitoring\brkunds{}a2a\brkunds{}count}
\item \textit{coll\brkunds{}monitoring\brkunds{}a2a\brkunds{}size}
\end{itemize}
Add to your command line at least \texttt{-{}-mca
  pml\brkunds{}monitoring\brkunds{}enable [1,2]} \\ Sequence of
\mpit{}:
\begin{enumerate}
\item {\texttt{MPI\brkunds{}T\brkunds{}init\brkunds{}thread}}
  Initialize the MPI\brkunds{}Tools interface
\item
  {\texttt{MPI\brkunds{}T\brkunds{}pvar\brkunds{}get\brkunds{}index}}
  To retrieve the variable id
\item {\texttt{MPI\brkunds{}T\brkunds{}session\brkunds{}create}} To
  create a new context in which you use your variable
\item {\texttt{MPI\brkunds{}T\brkunds{}handle\brkunds{}alloc}} To bind
  your variable to the proper session and MPI object
\item {\texttt{MPI\brkunds{}T\brkunds{}pvar\brkunds{}start}} To start
  the monitoring
\item Now you do all the communications you want to monitor
\item {\texttt{MPI\brkunds{}T\brkunds{}pvar\brkunds{}stop}} To stop
  and flush the monitoring
\item
  {\texttt{MPI\brkunds{}T\brkunds{}pvar\brkunds{}handle\brkunds{}free}}
\item
  {\texttt{MPI\brkunds{}T\brkunds{}pvar\brkunds{}session\brkunds{}free}}
\item {\texttt{MPI\brkunds{}T\brkunds{}finalize}}
\end{enumerate}

\subsection{Use of \textsc{LD\brkunds{}PRELOAD}}
\label{subsec:ldpreload}

In order to automatically generate communication matrices, you can use
the {\it monitoring\brkunds{}prof} tool that can be found in
\textit{test/monitoring/monitoring\brkunds{}prof.c}. While launching
your application, you can add the following option in addition to the
\texttt{-{}-mca pml\brkunds{}monitoring\brkunds{}enable} parameter:
\begin{description}
\item [\texttt{-x
    LD\_PRELOAD=ompi\_install\_dir/lib/monitoring\_prof.so}]
\end{description}

This library automatically gathers sent and received data into one
communication matrix. Although, the use of monitoring \mpit{} within
the code may interfere with this library. The main goal of this
library is to avoid dumping one file per MPI process, and gather
everything in one file aggregating all pieces of information.

The resulting communication matrices are as close as possible as the
effective amount of data exchanged between nodes. But it has to be
kept in mind that because of the stack of the logical layers in
\ompi{}, the amount of data recorded as part of collectives or
one-sided operations may be duplicated when the PML layer handles the
communication. For an exact measure of communications, the application
must use \mpit{}'s monitoring performance variables to potentially
subtract double-recorded data.

\subsection{Examples}

First is presented an example of monitoring using the \mpit{} in order
to define phases during which the monitoring component is active. A
second snippet is presented for how to access monitoring performance
variables with \mpit{}.

\subsubsection{Monitoring Phases}

You can execute the following example with
\\ \verb|mpiexec -n 4 --mca pml_monitoring_enable 2 test_monitoring|. Please
note that you need the prof directory to already exists to retrieve
the dumped files. Following the complete code example, you will find a
sample dumped file and the corresponding explanations.

\paragraph{test\_monitoring.c} (extract)

\begin{verbatim}
#include <stdlib.h>
#include <stdio.h>
#include <mpi.h>

static const void* nullbuff = NULL;
static MPI_T_pvar_handle flush_handle;
static const char flush_pvar_name[] = "pml_monitoring_flush";
static const char flush_cvar_name[] = "pml_monitoring_enable";
static int flush_pvar_idx;

int main(int argc, char* argv[])
{
  int rank, size, n, to, from, tagno, MPIT_result, provided, count;
  MPI_T_pvar_session session;
  MPI_Status status;
  MPI_Comm comm = MPI_COMM_WORLD;
  MPI_Request request;
  char filename[1024];

  /* Initialization of parameters */

  n = -1;
  MPI_Init(&argc, &argv);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_Comm_size(MPI_COMM_WORLD, &size);
  to = (rank + 1) % size;
  from = (rank + size - 1) % size;
  tagno = 201;

  /* Initialization of performance variables */

  MPIT_result = MPI_T_init_thread(MPI_THREAD_SINGLE, &provided);
  if (MPIT_result != MPI_SUCCESS)
    MPI_Abort(MPI_COMM_WORLD, MPIT_result);

  MPIT_result = MPI_T_pvar_get_index(flush_pvar_name, 
                                     MPI_T_PVAR_CLASS_GENERIC, 
                                     &flush_pvar_idx);
  if (MPIT_result != MPI_SUCCESS) {
    printf("cannot find monitoring MPI_T \"%s\" pvar, "
           "check that you have monitoring pml\n",
           flush_pvar_name);
    MPI_Abort(MPI_COMM_WORLD, MPIT_result);
  }

  MPIT_result = MPI_T_pvar_session_create(&session);
  if (MPIT_result != MPI_SUCCESS) {
    printf("cannot create a session for \"%s\" pvar\n", 
           flush_pvar_name);
    MPI_Abort(MPI_COMM_WORLD, MPIT_result);
  }

  /* Allocating a new PVAR in a session will reset the counters */

  MPIT_result = MPI_T_pvar_handle_alloc(session, flush_pvar_idx,
                                        &comm, 
                                        &flush_handle, 
                                        &count);
  if (MPIT_result != MPI_SUCCESS) {
    printf("failed to allocate handle on \"%s\" pvar, "
           "check that you have monitoring pml\n",
           flush_pvar_name);
    MPI_Abort(MPI_COMM_WORLD, MPIT_result);
  }

  /* First phase: make a token circulated in MPI_COMM_WORLD */

  MPIT_result = MPI_T_pvar_start(session, flush_handle);
  if (MPIT_result != MPI_SUCCESS) {
    printf("failed to start handle on \"%s\" pvar, "
           "check that you have monitoring pml\n",
           flush_pvar_name);
    MPI_Abort(MPI_COMM_WORLD, MPIT_result);
  }

  if (rank == 0) {
    n = 25;
    MPI_Isend(&n,1,MPI_INT,to,tagno,MPI_COMM_WORLD,&request);
  }
  while (1) {
    MPI_Irecv(&n, 1, MPI_INT, from, tagno, MPI_COMM_WORLD, &request);
    MPI_Wait(&request, &status);
    if (rank == 0) {n--;tagno++;}
    MPI_Isend(&n, 1, MPI_INT, to, tagno, MPI_COMM_WORLD, &request);
    if (rank != 0) {n--;tagno++;}
    if (n<0){
      break;
    }
  }

  /* 
   * Build one file per processes
   * Every thing that has been monitored by each
   * process since the last flush will be output in filename
   *
   * Requires directory prof to be created.
   * Filename format should display the phase number
   * and the process rank for ease of parsing with
   * aggregate_profile.pl script
   */

  sprintf(filename,"prof/phase_1");
  if( MPI_SUCCESS != MPI_T_pvar_write(session, flush_handle, 
                                      filename) ) 
  {
    fprintf(stderr, 
            "Process %d cannot save monitoring in %s.%d.prof\n", 
            rank, filename, rank);
  }

  /* Force the writing of the monitoring data */

  MPIT_result = MPI_T_pvar_stop(session, flush_handle);
  if (MPIT_result != MPI_SUCCESS) {
    printf("failed to stop handle on \"%s\" pvar, "
           "check that you have monitoring pml\n",
           flush_pvar_name);
    MPI_Abort(MPI_COMM_WORLD, MPIT_result);
  }

  /* 
   * Don't set a filename. If we stop the session before setting
   * it, then no output will be generated.
   */

  if( MPI_SUCCESS != MPI_T_pvar_write(session, flush_handle,
                                      &nullbuff) )
  {
    fprintf(stderr, 
            "Process %d cannot save monitoring in %s\n", 
            rank, filename);
  }

  (void)MPI_T_finalize();

  MPI_Finalize();
  
  return EXIT_SUCCESS;
}
\end{verbatim}

\paragraph{prof/phase\_1.0.prof}

\begin{verbatim}
# POINT TO POINT
E       0       1       108 bytes       27 msgs sent    0,0,0,27,0,[...],0
# OSC
# COLLECTIVES
D       MPI_COMM_WORLD  procs: 0,1,2,3
O2A     0       0 bytes 0 msgs sent
A2O     0       0 bytes 0 msgs sent
A2A     0       0 bytes 0 msgs sent
\end{verbatim}

As it show on the sample profiling, for each kind of communication
(point-to-point, one-sided and collective), you find all the related
informations. There is one line per peers communicating. Each line
start with a lettre describing the kind of communication, such as
follows:

\begin{description}
\item [{\tt E}] External messages, i.e. issued by the user
\item [{\tt I}] Internal messages, i.e. issued by the library
\item [{\tt S}] Sent one-sided messages, i.e. writing access to the remote memory
\item [{\tt R}] Received one-sided messages, i.e. reading access to the remote memory
\item [{\tt C}] Collective messages
\end{description}

This letter is followed by the rank of the issuing process, and the
rank of the receiving one. Then you have the total amount in bytes
exchanged and the count of messages. For point-to-point entries
(i.e. {\tt E} of {\tt I} entries), the line is completed by the full
distribution of messages in the form of a histogram. See variable {\tt
  size\brkunds{}histogram} in
Section~\ref{subsubsec:TDI-common-monitoring} for the corresponding
values. In the case of a disabled filtering between external and
internal messages, the {\tt I} lines are merged with the {\tt E}
lines, keeping the {\tt E} header.

The end of the summary is a per communicator information, where you
find the name of the communicator, the ranks of the processes included
in this communicator, and the amount of data send (or received) for
each kind of collective, with the corresponding count of operations of
each kind.

\subsubsection{Accessing Monitoring Performance Variables}

The following snippet presents how to access the performances
variables defined as part of the \mpit{} interface. The session
allocation is not presented as it is the same as in the previous
example. Please note that contrary to the {\it
  pml\brkunds{}monitoring\brkunds{}flush} variable, the class of the
monitoring performance values is {\tt
  MPI\brkunds{}T\brkunds{}PVAR\brkunds{}CLASS\brkunds{}SIZE}, whereas
the {\it flush} variable is of class {\tt GENERIC}. Also, performances
variables are only to be read.

\paragraph{test/monitoring/example\_reduce\_count.c} (extract)

\begin{verbatim}
MPI_T_pvar_handle count_handle;
int count_pvar_idx;
const char count_pvar_name[] = "pml_monitoring_messages_count";
size_t*counts;

/* Retrieve the proper pvar index */
MPIT_result = MPI_T_pvar_get_index(count_pvar_name, MPI_T_PVAR_CLASS_SIZE, &count_pvar_idx);
if (MPIT_result != MPI_SUCCESS) {
    printf("cannot find monitoring MPI_T \"%s\" pvar, check that you have monitoring pml\n",
           count_pvar_name);
    MPI_Abort(MPI_COMM_WORLD, MPIT_result);
}

/* Allocating a new PVAR in a session will reset the counters */
MPIT_result = MPI_T_pvar_handle_alloc(session, count_pvar_idx,
                                      &comm, &count_handle, &count);
if (MPIT_result != MPI_SUCCESS) {
    printf("failed to allocate handle on \"%s\" pvar, check that you have monitoring pml\n",
           count_pvar_name);
    MPI_Abort(MPI_COMM_WORLD, MPIT_result);
}

counts = (size_t*)malloc(count * sizeof(size_t));

MPIT_result = MPI_T_pvar_start(session, count_handle);
if (MPIT_result != MPI_SUCCESS) {
    printf("failed to start handle on \"%s\" pvar, check that you have monitoring pml\n",
           count_pvar_name);
    MPI_Abort(MPI_COMM_WORLD, MPIT_result);
}

/* Token Ring communications */
if (rank == 0) {
    n = 25;
    MPI_Isend(&n,1,MPI_INT,to,tagno,MPI_COMM_WORLD,&request);
}
while (1) {
    MPI_Irecv(&n, 1, MPI_INT, from, tagno, MPI_COMM_WORLD, &request);
    MPI_Wait(&request, &status);
    if (rank == 0) {n--;tagno++;}
    MPI_Isend(&n, 1, MPI_INT, to, tagno, MPI_COMM_WORLD, &request);
    if (rank != 0) {n--;tagno++;}
    if (n<0){
        break;
    }
}

MPIT_result = MPI_T_pvar_read(session, count_handle, counts);
if (MPIT_result != MPI_SUCCESS) {
    printf("failed to read handle on \"%s\" pvar, check that you have monitoring pml\n",
           count_pvar_name);
    MPI_Abort(MPI_COMM_WORLD, MPIT_result);
}

/* Global reduce so everyone knows the maximum messages sent to each rank */
MPI_Allreduce(MPI_IN_PLACE, counts, count, MPI_UNSIGNED_LONG, MPI_MAX, MPI_COMM_WORLD);

/* OPERATIONS ON COUNTS */
...

free(counts);

MPIT_result = MPI_T_pvar_stop(session, count_handle);
if (MPIT_result != MPI_SUCCESS) {
    printf("failed to stop handle on \"%s\" pvar, check that you have monitoring pml\n",
           count_pvar_name);
    MPI_Abort(MPI_COMM_WORLD, MPIT_result);
}

MPIT_result = MPI_T_pvar_handle_free(session, &count_handle);
if (MPIT_result != MPI_SUCCESS) {
    printf("failed to free handle on \"%s\" pvar, check that you have monitoring pml\n",
           count_pvar_name);
    MPI_Abort(MPI_COMM_WORLD, MPIT_result);
}
\end{verbatim}

\section{Technical Documentation of the Implementation}
\label{sec:TDI}

This section describes the technical details of the components
implementation. It is of no use from a user point of view but it is made
to facilitate the work for future developer that would debug or enrich
the monitoring components.

The architecture of this component is as follows. The Common component
is the main part where the magic occurs. PML, OSC and COLL components
are the entry points to the monitoring tool from the software stack
point-of-view. The relevant files can be found in accordance with
the partial directory tree presented in Figure~\ref{fig:tree}.

\begin{figure}
  \dirtree{%
    .1 ompi/mca/. 
    .2 common. 
    .3 monitoring.
    .4 common\_monitoring.h. 
    .4 common\_monitoring.c.
    .4 common\_monitoring\_coll.h. 
    .4 common\_monitoring\_coll.c.   
    .4 HowTo\_pml\_monitoring.tex. 
    .4 Makefile.am.
    .2 pml. 
    .3 monitoring. 
    .4 pml\_monitoring.h.
    .4 pml\_monitoring\_component.c.
    .4 pml\_monitoring\_comm.c. 
    .4 pml\_monitoring\_irecv.c. 
    .4 pml\_monitoring\_isend.c. 
    .4 pml\_monitoring\_start.c. 
    .4 pml\_monitoring\_iprobe.c.
    .4 Makefile.am.
    .2 osc. 
    .3 monitoring.
    .4 osc\_monitoring.h. 
    .4 osc\_monitoring\_component.c.
    .4 osc\_monitoring\_comm.h. 
    .4 osc\_monitoring\_module.h.
    .4 osc\_monitoring\_dynamic.h. 
    .4 osc\_monitoring\_template.h.
    .4 osc\_monitoring\_accumulate.h. 
    .4 osc\_monitoring\_active\_target.h.
    .4 osc\_monitoring\_passive\_target.h.
    .4 configure.m4.
    .4 Makefile.am.
    .2 coll. 
    .3 monitoring.
    .4 coll\_monitoring.h. 
    .4 coll\_monitoring\_component.c.
    .4 coll\_monitoring\_bcast.c. 
    .4 coll\_monitoring\_reduce.c.
    .4 coll\_monitoring\_barrier.c. 
    .4 coll\_monitoring\_alltoall.c.
    .4 {...} .
    .4 Makefile.am.
  }
\caption{Monitoring component files architecture (partial)}
\label{fig:tree}
\end{figure}

\subsection{Common}
\label{subsec:TDI-common}
This part of the monitoring components is the place where data is
managed. It centralizes all recorded information, the translation
hash-table and ensures a unique initialization of the monitoring
structures. This component is also the one where the MCA variables (to
be set as part of the command line) are defined and where the final
output, if any requested, is dealt with.

The header file defines the unique monitoring version number,
different preprocessing macros for printing information using the
monitoring output stream object, and the ompi monitoring API (i.e. the
API to be used INSIDE the ompi software stack, not the one to be
exposed to the end-user). It has to be noted that the {\tt
  mca\brkunds{}common\brkunds{}monitoring\brkunds{}record\brkunds{}*}
functions are to be used with the destination rank translated into the
corresponding rank in {\tt MPI\brkunds{}COMM\brkunds{}WORLD}. This
translation is done by using {\tt
  mca\brkunds{}common\brkunds{}monitoring\brkunds{}get\brkunds{}world\brkunds{}rank}. The
use of this function may be limited by how the initialization occurred
(see in~\ref{subsec:TDI-pml}).

\subsubsection{Common monitoring}
\label{subsubsec:TDI-common-monitoring}

The the common\brkunds{}monitoring.c file defines multiples variables
that has the following use:
\begin{description}
\item[{\tt mca\brkunds{}common\brkunds{}monitoring\brkunds{}hold}] is
  the counter that keeps tracks of whether the common component has
  already been initialized or if it is to be released. The operations
  on this variable are atomic to avoid race conditions in a
  multi-threaded environment.
\item[{\tt
    mca\brkunds{}common\brkunds{}monitoring\brkunds{}output\brkunds{}stream\brkunds{}obj}]
  is the structure used internally by \ompi{} for output streams. The
  monitoring output stream states that this output is for debug, so
  the actual output will only happen when OPAL is configured with {\tt
    -{}-enable-debug}. The output is sent to stderr standard output
  stream. The prefix field, initialized in {\tt
    mca\brkunds{}common\brkunds{}monitoring\brkunds{}init}, states
  that every log message emitted from this stream object will be
  prefixed by ``{\tt [hostname:PID] monitoring: }'', where {\tt
    hostname} is the configured name of the machine running the
  process and {\tt PID} is the process id, with 6 digits, prefixed
  with zeros if needed.
\item[{\tt mca\brkunds{}common\brkunds{}monitoring\brkunds{}enabled}]
  is the variable retaining the original value given to the MCA option
  system, as an example as part of the command line. The corresponding
  variable is {\tt pml\brkunds{}monitoring\brkunds{}enable}. This
  variable is not to be written by the monitoring component. It is
  used to reset the {\tt
    mca\brkunds{}common\brkunds{}monitoring\brkunds{}current\brkunds{}state}
  variable between phases. The value given to this parameter also
  defines whether or not the filtering between internal and externals
  messages is enabled.
\item[{\tt
    mca\brkunds{}common\brkunds{}monitoring\brkunds{}current\brkunds{}state}]
  is the variable used to determine the actual current state of the
  monitoring. This variable is the one used to define phases.
\item[{\tt
    mca\brkunds{}common\brkunds{}monitoring\brkunds{}output\brkunds{}enabled}]
  is a variable, set by the MCA engine, that states whether or not the
  user requested a summary of the monitored data to be streamed out at
  the end of the execution. It also states whether the output should
  be to stdout, stderr or to a file. If a file is requested, the next
  two variables have to be set. The corresponding variable is {\tt
    pml\brkunds{}monitoring\brkunds{}enable\brkunds{}output}. {\bf
    Warning:} This variable may be set to 0 in case the monitoring is
  also controlled with \mpit{}. We cannot both control the monitoring
  via \mpit{} and expect accurate answer upon {\tt
    MPI\brkunds{}Finalize}.
\item[{\tt
    mca\brkunds{}common\brkunds{}monitoring\brkunds{}initial\brkunds{}filename}]
  works the same as {\tt
    mca\brkunds{}common\brkunds{}monitoring\brkunds{}ena\allowbreak{}bled}. This
  variable is, and has to be, only used as a placeholder for the {\tt
    pml\brkunds{}monitoring\allowbreak\brkunds{}filename}
  variable. This variable has to be handled very carefully as it has
  to live as long as the program and it has to be a valid pointer
  address, which content is not to be released by the component. The
  way MCA handles variable (especially strings) makes it very easy to
  create segmentation faults. But it deals with the memory release of
  the content. So, in the end, {\tt
    mca\brkunds{}common\brkunds{}monitoring\brkunds{}initial\brkunds{}filename}
  is just to be read.
\item[{\tt
    mca\brkunds{}common\brkunds{}monitoring\brkunds{}current\brkunds{}filename}]
  is the variable the monitoring component will work with. This
  variable is the one to be set by \mpit{'s} control variable {\tt
    pml\brkunds{}monitoring\brkunds{}flush}. Even though this control
  variable is prefixed with {\tt pml} for historical and easy reasons,
  it depends on the common section for its behavior.
\item[{\tt pml\brkunds{}data} and {\tt pml\brkunds{}count}] arrays of
  unsigned 64-bits integers record respectively the cumulated amount
  of bytes sent from the current process to another process $p$, and
  the count of messages. The data in this array at the index $i$
  corresponds to the data sent to the process $p$, of id $i$ in {\tt
    MPI\brkunds{}COMM\brkunds{}WORLD}. These arrays are of size $N$,
  where $N$ is the number of nodes in the MPI application. If the
  filtering is disabled, these variables gather all information
  regardless of the tags. In this case, the next two arrays are,
  obviously, not used, even though they will still be allocated. The
  {\tt pml\brkunds{}data} and {\tt pml\brkunds{}count} arrays, and the
  nine next arrays described, are allocated, initialized, reset and
  freed all at once, and are concurrent in the memory.
\item[{\tt filtered\brkunds{}pml\brkunds{}data} and {\tt
    filtered\brkunds{}pml\brkunds{}count}] arrays of unsigned 64-bits
  integers record respectively the cumulated amount of bytes sent from
  the current process to another process $p$, and the count of
  internal messages. The data in this array at the index $i$
  corresponds to the data sent to the process $p$, of id $i$ in {\tt
    MPI\brkunds{}COMM\brkunds{}WORLD}. These arrays are of size $N$,
  where $N$ is the number of nodes in the MPI application. The
  internal messages are defined as messages sent through the PML
  layer, with a negative tag. They are issued, as an example, from the
  decomposition of collectives operations.
\item[{\tt osc\brkunds{}data\brkunds{}s} and {\tt
    osc\brkunds{}count\brkunds{}s}] arrays of unsigned 64-bits
  integers record respectively the cumulated amount of bytes sent from
  the current process to another process $p$, and the count of
  messages. The data in this array at the index $i$ corresponds to the
  data sent to the process $p$, of id $i$ in {\tt
    MPI\brkunds{}COMM\brkunds{}WORLD}. These arrays are of size $N$,
  where $N$ is the number of nodes in the MPI application.
\item[{\tt osc\brkunds{}data\brkunds{}r} and {\tt
    osc\brkunds{}count\brkunds{}r}] arrays of unsigned 64-bits
  integers record respectively the cumulated amount of bytes received
  to the current process to another process $p$, and the count of
  messages. The data in this array at the index $i$ corresponds to the
  data sent to the process $p$, of id $i$ in {\tt
    MPI\brkunds{}COMM\brkunds{}WORLD}. These arrays are of size $N$,
  where $N$ is the number of nodes in the MPI application.
\item[{\tt coll\brkunds{}data} and {\tt coll\brkunds{}count}] arrays
  of unsigned 64-bits integers record respectively the cumulated
  amount of bytes sent from the current process to another process
  $p$, in the case of a all-to-all or one-to-all operations, or
  received from another process $p$ to the current process, in the
  case of all-to-one operations, and the count of messages. The data
  in this array at the index $i$ corresponds to the data sent to the
  process $p$, of id $i$ in {\tt
    MPI\brkunds{}COMM\brkunds{}WORLD}. These arrays are of size $N$,
  where $N$ is the number of nodes in the MPI application. The
  communications are thus considered symmetrical in the resulting
  matrices.
\item[{\tt size\brkunds{}histogram}] array of unsigned 64-bits
  integers records the distribution of sizes of pml messages, filtered
  or not, between the current process and a process $p$. This
  histogram is of log-2 scale. The index 0 is for empty
  messages. Messages of size between 1 and $2^{64}$ are recorded such
  as the following. For a given size $S$, with $2^k \le S < 2^{k+1}$,
  the $k$-th element of the histogram is incremented. This array is of
  size $N \times {\tt max\brkunds{}size\brkunds{}histogram}$, where
  $N$ is the number of nodes in the MPI application.
\item[{\tt max\brkunds{}size\brkunds{}histogram}] constant value
  correspond to the number of elements in the {\tt
    size\brkunds{}histo\allowbreak{}gram} array for each processor. It
  is stored here to avoid having its value hang here and there in the
  code. This value is used to compute the total size of the array to
  be allocated, initialized, reset or freed. This value equals $(10 +
  {\tt max\brkunds{}size\brkunds{}histogram}) \times N$, where $N$
  correspond to the number of nodes in the MPI application. This value
  is also used to compute the index to the histogram of a given
  process $p$ ; this index equals $i \times {\tt
    max\brkunds{}size\brkunds{}histogram}$, where $i$ is $p$'s id in
  {\tt MPI\brkunds{}COMM\brkunds{}WORLD}.
\item[{\tt log10\brkunds{}2}] is a cached value for the common
  logarithm (or decimal logarithm) of 2. This value is used to compute
  the index at which increment the histogram value. This index $j$,
  for a message that is not empty, is computed as follow $j = 1 +
  \left \lfloor{log_{10}(S)/log_{10}(2)} \right \rfloor$, where
  $log_{10}$ is the decimal logarithm and $S$ the size of the message.
\item[{\tt rank\brkunds{}world}] is the cached value of the current
  process in {\tt MPI\brkunds{}COMM\brkunds{}WORLD}.
\item[{\tt nprocs\brkunds{}world}] is the cached value of the size of
  {\tt MPI\brkunds{}COMM\brkunds{}WORLD}.
\item[{\tt
    common\brkunds{}monitoring\brkunds{}translation\brkunds{}ht}] is
  the hash table used to translate the rank of any process $p$ of rank
  $r$ from any communicator, into its rank in {\tt
    MPI\brkunds{}COMM\brkunds{}WORLD}. It lives as long as the
  monitoring components do.
\end{description}

In any case, we never monitor communications between one process and
itself.

The different functions to access \mpit{} performance variables are
pretty straight forward. Note that for PML, OSC and COLL, for both
count and size, performance variables the {\it notify} function is the
same. At binding, it sets the {\tt count} variable to the size of {\tt
  MPI\brkunds{}COMM\brkunds{}WORLD}, as requested by the MPI-3
standard (for arrays, the parameter should be set to the number of
elements of the array). Also, the {\it notify} function is responsible
for starting the monitoring when any monitoring performance value
handle is started, and it also disable the monitoring when any
monitoring performance value handle is stopped. The {\it flush}
control variable behave as follows. On binding, it returns the size of
the filename defined if any, 0 otherwise. On start event, this
variable also enable the monitoring, as the performance variables do,
but it also disable the final output, even though it was previously
requested by the end-user. On the stop event, this variable flushes
the monitored data to the proper output stream (i.e. stdout, stderr or
the requested file). Note that these variables are to be bound only
with the {\tt MPI\brkunds{}COMM\brkunds{}WORLD} communicator. For far,
the behavior in case of a binding to another communicator is not
tested.

For the flushing itself, it is decomposed into two functions. The
first one ({\tt
  mca\brkunds{}common\brkunds{}monitoring\brkunds{}flush}) is
responsible for opening the proper stream. If it is given 0 as its
first parameter, it does nothing with no error propagated as it
correspond to a disable monitoring. The {\tt filename} parameter is
only taken in account if {\tt fd} is strictly greater than 2. Note
that upon flushing, the record arrays are reset to 0. Also, the
flushing called in {\it common\brkunds{}monitoring.c} call the
specific flushing for per communicator collectives monitoring data.

For historical reasons, and because of the fact that the PML layer is
the first one to be loaded, MCA parameters and the {\it
  monitoring\brkunds{}flush} control variable are linked to the PML
framework. The other performance variables, though, are linked to the
proper frameworks.

\subsubsection{Common Coll Monitoring}
\label{subsubsec:TDI-common-coll}

In addition to the monitored data kept in the arrays, the monitoring
component also provide a per communicator set of records. It keeps
pieces of information about collective operations. As we cannot know
how the data are indeed exchanged (see Section~\ref{subsec:TDI-coll}),
we added this complement to the final summary of the monitored
operations.

We keep the per communicator data set as part of the {\it
  coll\brkunds{}monitoring\brkunds{}module}. Each data set is also
kept in a hash table, with the communicator structure address as the
hash-key. This data set is made to keep tracks of the mount of data
sent through a communicator with collective operations and the count
of each kind of operations. It also cache the list of the processes'
ranks, translated to their rank in {\tt
  MPI\brkunds{}COMM\brkunds{}WORLD}, as a string, the rank of the
current process, translated into its rank in {\tt
  MPI\brkunds{}COMM\brkunds{}WORLD} and the communicator's name.

The process list is generated with the following algorithm. First, we
allocate a string long enough to contain it. We define long enough as
$1 + (d + 2) \times s$, where $d$ is the number of digit of the higher
rank in {\tt MPI\brkunds{}COMM\brkunds{}WORLD} and $s$ the size of the
current communicator. We add 2 to $d$, to consider the space needed
for the comma and the space between each rank, and 1 to ensure there
is enough room for the NULL character terminating the string. Then, we
fill the string with the proper values, and adjust the final size of
the string.

When possible, this process happen when the communicator is being
created. If it fails, this process will be tested again when the
communicator is being released.

This data set lifetime is different from the one of its corresponding
communicator. It is actually destroyed only once its data had been
flushed (at the end of the execution or at the end of a monitoring
phase). To this end, this structure keeps a flag to know if it is safe
to release it or not.

\subsection{PML}
\label{subsec:TDI-pml}

As specified in Section~\ref{subsubsec:TDI-common-monitoring}, this
component is closely working with the common component. They were
merged initially, but separated later in order to propose a cleaner
and more logical architecture.

This module is the first one to be initialized by the \ompi{} software
stack ; thus it is the one responsible for the proper initialization,
as an example, of the translation hash table. \ompi{} relies on the
PML layer to add process logical structures as far as communicators
are concerned.

To this end, and because of the way the PML layer is managed by the
MCA engine, this component has some specific variables to manage its
own state, in order to be properly instantiated. The module selection
process works as follows. All the PML modules available for the
framework are loaded, initialized and asked for a priority. The higher
the priority, the higher the odds to be selected. This is why our
component returns a priority of 0. Note that the priority is returned
and initialization of the common module is done at this point only if
the monitoring had been requested by the user.

% CF - TODO: check what happen if the monitoring is the only PML module available.
If everything works properly, we should not be selected. The next step
in the PML initialization is to finalize every module that is not the
selected one, and then close components that were not used. At this
point the winner component and its module are saved for the PML. The
variables {\tt
  mca\brkunds{}pml\brkunds{}base\brkunds{}selected\brkunds{}component}
and {\tt mca\brkunds{}pml}, defined in {\it
  ompi/mca/pml/base/pml\brkunds{}base\brkunds{}frame.c}, are now
initialized. This point is the one where we install our interception
layer. We also indicate ourself now initialized, in order to know on
the next call to the {\it component\brkunds{}close} function that we
actually have to be closed this time. Note that the adding of our
layer require the add of the {\tt
  MCA\brkunds{}PML\brkunds{}BASE\brkunds{}FLAG\brkunds{}REQUIRE\brkunds{}WORLD}
flag in order to request for the whole list of processes to be given
at the initialization of {\tt MPI\brkunds{}COMM\brkunds{}WORLD}, so we
can properly fill our hash table. The downside of this trick is that
it stops the \ompi{} optimization of lazily adding them.

Once that is done, we are properly installed, and we can monitor every
messages going through the PML layer. As we only monitor messages from
the emitter side, we only actually record when the messages are using
the {\tt MPI\brkunds{}Send}, {\tt MPI\brkunds{}Isend} or {\tt
  MPI\brkunds{}Start} functions.

\subsection{OSC}
\label{subsec:TDI-osc}

This layer is responsible for remote memory access operations, and
thus, it has its specificities. Even though the component selection
process is quite close to the PML selection's one, there are some
aspects on the usage of OSC modules that had us to adapt the
interception layer.

The first problem comes from how the module is accessed inside the
components. In the OSC layer, the module is part of the {\tt
  ompi\brkunds{}win\brkunds{}t} structure. This implies that it is
possible to access directly to the proper field of the structure to
find the reference to the module. And it how it is done. Because of
that it is not possible to directly replace a module with ours that
would have saved the original module. The first solution was then to
``extend'' (in the ompi manner of extending {\it objects}) with a
structure that would have contain as the first field a union type of
every possible module. We would have then copy their fields values,
save their functions, and replace them with pointers to our inception
functions. This solution was implemented but a second problem was
faced, stopping us from going with this solution.

The second problem was that the {\it osc/rdma} uses internally a hash
table to keep tracks of its modules and allocated segments, with the
module's pointer address as the hash key. Hence, it was not possible
for us to modify this address, as the RDMA module would not be able to
find the corresponding segments. This also implies that it is neither
possible for us to extend the structures. Therefore, we could only
modify the common fields of the structures to keep our ``module''
adapted to any OSC component. We designed templates, dynamically
adapted for each kind of module.

To this end and for each kind of OSC module, we generate and
instantiate three variables:
\begin{description}
\item[{\tt
    OMPI\brkunds{}OSC\brkunds{}MONITORING\brkunds{}MODULE\brkunds{}VARIABLE(template)}]
  is the structure that keeps the address of the original module
  functions of a given component type (i.e. RDMA, PORTALS4, PT2PT or
  SM). It is initialized once, and referred to to propagate the calls
  after the initial interception. There is one generated for each kind
  of OSC component.
\item[{\tt
    OMPI\brkunds{}OSC\brkunds{}MONITORING\brkunds{}MODULE\brkunds{}INIT(template)}]
  is a flag to ensure the module variable is only initialized once, in
  order to avoid race conditions. There is one generated for each {\tt
    OMPI\brkunds{}OSC\brkunds{}MONITORING\brkunds{}MODULE\brkunds{}VARIABLE(template)},
  thus one per kind of OSC component.
\item[{\tt
    OMPI\brkunds{}OSC\brkunds{}MONITORING\brkunds{}TEMPLATE\brkunds{}VARIABLE(template)}]
  is a structure containing the address of the interception
  functions. There is one generated for each kind of OSC component.
\end{description}

The interception is done with the following steps. First, we follow
the selecting process. Our priority is set to {\tt INT\brkunds{}MAX}
in order to ensure that we would be the selected component. Then we do
this selection ourselves. This gives us the opportunity to modify as
needed the communication module. If it is the first time a module of
this kind of component is used, we extract from the given module the
function's addresses and save them to the {\tt
  OMPI\brkunds{}OSC\brkunds{}MONITORING\brkunds{}MODULE\brkunds{}VARIABLE(template)}
structure, after setting the initialization flag. Then we replace the
origin functions in the module with our interception ones.

To make everything work for each kind of component, the variables are
generated with the corresponding interception functions. These
operations are done at compilation time. An issue appeared with the
use of PORTALS4, that have its symbols propagated only when the card
are available on the system. In the header files, where we define the
template functions and structures, {\it template} refers to the OSC
component name.

We found two drawbacks to this solution. First, the readability of the
code is bad. Second, is that this solution is not auto-adaptive to new
components. If a new component is added, the code in {\it
  ompi/mca/osc/monitoring/osc\brkunds{}monitoring\brkunds{}component.c}
needs to be modified in order to monitor the operations going through
it. Even though the modification is three lines long, it my be
preferred to have the monitoring working without any modification
related to other components.

A second solution for the OSC monitoring could have been the use of a
hash table. We would have save in the hash table the structure
containing the original function's addresses, with the module address
as a hash key. Our interception functions would have then search in
the hash table the corresponding structure on every call, in order to
propagate the functions calls. This solution was not implemented
because because it offers an higher memory footprint for a large
amount of windows allocated. Also, the cost of our interceptions would
have been then higher, because of the search in the hash table. This
reason was the main reason we choose the first solution. The OSC layer
is designed to be very cost-effective in order to take the best
advantages of the background communication and
communication/computations overlap. This solution would have however
give us the adaptability our solution lacks.

\subsection{COLL}
\label{subsec:TDI-coll}

The collective module (or to be closer to the reality, {\it modules})
is part of the communicator. The modules selection is made with the
following algorithm. First all available components are selected,
queried and sorted in ascending order of priorities. The modules may
provide part or all operations, keeping in mind that modules with
higher priority may take your place. The sorted list of module is
iterated over, and for each module, for each operation, if the
function's address is not {\tt NULL}, the previous module is replace
with the current one, and so is the corresponding function. Every time
a module is selected it is retained and enabled (i.e. the {\tt
  coll\brkunds{}module\brkunds{}enable} function is called), and every
time it gets replaced, it is disabled (i.e. the {\tt
  coll\brkunds{}module\brkunds{}disable} function is called) and
released.

When the monitoring module is queried, the priority returned is {\tt
  INT\brkunds{}MAX} to ensure that our module comes last in the
list. Then, when enabled, all the previous function-module couples are
kept as part of our monitoring module. The modules are retained to
avoid having the module freed when released by the selecting
process. To ensure the error detection in communicator (i.e. an
incomplete collective API), if, for a given operation, there is no
corresponding module given, we set this function's address to {\tt
  NULL}. Symmetrically, when our module is released, we also propagate
this call to each underlying module, and we also release the
objects. Also, when the module is enabled, we initialize the per
communicator data record, which gets released when the module is
disabled.

When an collective operation is called, both blocking or non blocking,
we intercept the call and record the data in two different
entries. The operations are groups between three kinds. One-to-all
operations, all-to-one operations and all-to-all operations.

For one-to-all operations, the root process of the operation computes
the total amount of data to be sent, and keep it as part of the per
communicator data (see Section~\ref{subsubsec:TDI-common-coll}). Then
it update the {\it common\brkunds{}monitoring} array with the amount
of data each pair has to receive in the end. As we cannot predict the
actual algorithm used to communicate the data, we assume the root send
everything directly to each process.

For all-to-one operations, each non-root process compute the amount of
data to send to the root and update the {\it common\brkunds{}monitoring}
array with the amount of data at the index $i$, with $i$ being the
rank in {\tt MPI\brkunds{}COMM\brkunds{}WORLD} of the root process. As
we cannot predict the actual algorithm used to communicate the data,
we assume each process send its data directly to the root. The root
process compute the total amount of data to receive and update the per
communicator data.

For all-to-all operations, each process compute for each other process
the amount of data to both send and receive from it. The amount of
data to be sent to each process $p$ is added to update the {\it
  common\brkunds{}monitoring} array at the index $i$, with $i$ being
the rank of $p$ in {\tt MPI\brkunds{}COMM\brkunds{}WORLD}. The total
amount of data sent by a process is also added to the per communicator
data.

For every rank translation, we use the {\tt
  common\brkunds{}monitoring\brkunds{}translation\brkunds{}ht} hash
table.

\end{document}
